Information theory

Information theory is concerned with two main tasks. The first task is called data compression (source coding). This is concerned with removing redundancy from data so it can be represented more compactly (either exactly, in a lossless way, or approximately, in a lossy way). The second task is error correction (channel coding), which means encoding data in such a way that it is robust to errors when sent over a noisy channel, such as a telephone line. It turns out that the amount by which you can compress data, and the amount of redundancy you need to add to a message before transmission, are both closely related to how predictable the source of data is, i.e., to its probability distribution. Hence there is a deep connection between information theory and statistics/ machine learning (indeed, there is an excellent book on this topic [Mac03]). In this chapter, we introduce some of the key information-theoretic measures of uncertainty and statistical dependence, such as entropy and mutual information.